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BACKGROUND OF THE INVENTION 

The invention relates to camera lens systems having 
optical characteristics (such as zoom and focus) that can be 
10 modified while the camera is in use; the lens system is for 
mounting on a camera that is 11 instrumented" , i.e. having its 
own sensors for delivering signals representative of the 
angular position of the camera relative to its stand. A 
particularly important, although not exclusive, application of 
15 the invention lies in lens systems for use with cameras that 
are incorporated in installations that are capable of 
replacing the representation of a target zone in the image 
supplied by the camera and occupying a fixed position in the 
space viewed by the camera with a stored "model" or pattern 
20 that can be still or animated, and with the model being scaled 
appropriately . ■ The position and the area of the fixed zone 
are stored in the installation which includes computation 
means and image synthesis means capable of computing the 
deformation or "warping" that need be imparted to the model, 
25 and capable of superimposing the warped model. 

In general, the sensors used are digital encoders having 
the advantage of suffering no drift and^ of delivering an 
output that is directly usable by data processing equipment. 

Installations of the above type already exist. To obtain 
30 satisfactory precision, implementation of such installations 
requires a lengthy and arduous calibration stage to be 
performed on site prior to shooting, during which a camera 
operator aims successively at various different portions of 
the entire space that can be scanned while using different 
35 degrees of zoom and a plurality of different focus settings, 
and the various sensors are calibrated from the shots using 
close and far fixed characteristic points in the observed 
scene as references. Those methods restrict the field of 
operation . 

40 Other parameter measurement systems are known and for 

instance reference may be made to the paper "A high precision 




camera operation parameter measurement system and its 
application to image motion inferring " by Ziieng et al, in 



The inventors have become aware that some of the transfer 
functions and characteristics to be identified are intrinsic 
to the lens system itself, whereas others depend on the 

10 interface' between the lens system and the camera and can be 
referred as being "extrinsic". In particular, a given lens 
will not always be in exactly the same position relative to 
the camera's focal plane that is occupied by the array of 
photosensitive sites, due to inevitable tolerance (wear) , 

15 assembly conditions, and image standards. 

On the basis of these observations, it is an object of the 
invention to simplify quite considerably the operations, that 
need to be performed on site each time a new camera or a new 
location is used. 

20 To do this, the invention provides in particular a 

calibration method comprising the steps of : 

- calibrating, once for all, intrinsic characteristics 
or parameters of the lens system and establishing a computer 
file containing said characteristics; and 

25 - on site, calibrating the assembled camera and lens 

system so as to define transfer functions between signals 
coming from camera orientation sensors and lens system sensors 
and real characteristics on the basis of said file and of 
signals obtained by shooting characteristic points of the 

30 scene to be observed. 

In another aspect, a calibration method is proposed 
enabling a correspondence table to be determined between at 
least the output signals from zoom and focusing sensors Z and 
F placed on a camera lens and constituted by digital coders 

35 with real values for focal length and geometrical deformation, 
the method comprising: 

(a) a step, that is performed once for all, of determining 
the intrinsic characteristics of the lens; and 



"Proceedings of International Conference on 
ICIP '99, Kobe, Japan, to which reference may 



Image processing, 
be made . 
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SUMMARY OF THE INVENTION 



(b) a step, that is performed on site after the lens has 
been mounted on a camera for a particular purpose, of 
resetting origins. 

This greatly simplifies on-site operations since' they can 
be reduced to taking a few shots that can <be performed in a 
few minutes by an operator who has been trained but is not a 
specialist, with all of the intrinsic characteristics of the 
lens system being already available. The lens system 

calibration file need merely be selected, the constants of the 
system (type of camera head, position of camera on camera 
head) be given, and then the necessary shots be taken. In 
addition, this method provides great accuracy since intrinsic 
calibration is performed free from operating constraints. 

The invention also proposes an assembly or package that 
can be used for use of a camera lens which constitutes said 
lens system, the assembly comprising the lens itself plus a 
file (ROM file, flash memory, disk) accompanying it and 
loadable into a computer used during the on-site portion of 
calibration. 

The invention also provides software that can be executed 
on a computer and that includes a database containing the 
intrinsic characteristics of a lens system and a program 
responsive to the database and to measurements performed on- 
site while the lens system is mounted on .a, camera to determine 
the extrinsic characteristics and the complete two-dimensional 
(2D) transfer function between the scene as seen by the camera 
and the displayed image. 

In another aspect of the invention, there is provided a 
calibration method enabling a correspondence table to be 
established between at least the output signals from zoom and 
focus sensors Z and F placed on a camera lens and constituted 
by digital coders, and real values of focal length and 
geometrical deformation for the camera on which the lens 
system is mounted, the method comprising: 

(a) a stage , performed once and for always, of determining 
the intrinsic characteristics of the lens, which stage is 
performed after the lens has been mounted on a camera, and 
during which the following steps are performed: 



- taking a plurality of shots with the camera in 
different pan and tilt orientations and different zoom and 
focus values (where focus influences focal length) ; 

- for each shot, reading output signals from the 
coders and the positions in the image of at least two ooints, 
a near point and a far point, in the scene observed in the 
shot; and 

drawing up an intrinsic calibration table by 
comparing the signal values and the positions of the points in 
the image provided by the camera; and 

(b) a stage performed on site after the lens has been 
mounted on a camera that is to be used on site, during which 
stage, the following steps are performed: 

- specifying operating conditions (for example: PAL, NTSC, 
. . , , standard; 4/3 or 16/9 image ratio, high definition, . . . 
image size; camera position on the camera head after 
balancing, etc.); and 

- repeating only some of the operations performed in stage 
(a) , solely insofar as they are necessary for resetting 
origins and for determining parameters including the pixel 
aspect ratio. 

The near point observed during stage (a) could be provided 
by a point source such, as a light emitting diode (LED) or a 
laser diode placed at a distance that is slightly greater than 
the nearest focus distance. The far point must simulate 
viewing at infinity and is preferably at least 100 meters (m) 
away . 

In general, all of the measurements in stage (a) should be 
performed prior to performing the computations which are 
subsequently performed globally either on site or remotely. 
For example, these measurements can be performed using a. 
program that is well defined as a plurality of steps, each 
step being performed with the same zoom setting, and a 
plurality of different focus settings. 

The independence between the zoom and focus 
characteristics then makes it possible to represent the lens 
by a mathematical model in the form of two tables representing 
two families of functions each having a single input variable, 
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whereas an approach without separation would have required a 
table with two input variables which would make interpretation 
difficult. In other words, the lens is represented by a 
mathematical model using single input variable functions only. 
5 An additional advantage of . the ' method is 'that it makes 

remote calibration possible so there is ' no 'need to disclose 
the calibration software, and that would-be impossible without 
the last two steps of stage (a) being separated. An 
application server can be provided using the SMTP protocol, 
10 while the calibration machine is connected temporarily to the 
Internet. For the purposes of an Internet connection, the 
data obtained by the "snaps" mentioned below are sent to a 
.server which responds by sending a calibration file which can 
optionally be encrypted. 
15 The camera fitted with the lens can be associated with at 

least one gyrosensor (gyro or rate gyro) which provides angle • 
measurements in addition to those supplied by the camera head 
sensors, e.g. when the camera head is subjected to vibration 
(as in a stadium) . The gyro is for measuring tilt for which 
20 out-of -balance weight can give rise to measurement errors. 
The camera is balanced statically but not dynamically: so long 
as it is not in motion, it is balanced, but under the' effect 
of excitation (for the most part along the vertical axis if* 
the structure is sound) translation motion gives rise to tilt 
25 rotation. On a structure having resonant modes that are more 
troublesome, for example horizontal translation motion, it is 
possible to envisage a second gryosensor responsive to pan. 

The data originating from the sensors representing the 
state of the instrumented camera can be used directly in 
30 conventional manner or can be converted into an audio 
frequency signal, thereby making it possible to use the 
ordinary audio-visual environment for data transport and/or 
recording . 

The above characteristics and others will appear more 
35 clearly on reading the following description of a particular 
embodiment given by way of non-limiting example. The 
description refers to the accompanying drawings. 



BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a diagram showing the functions of the 
various sensors carried by the lens and the head of 'an 
instrumented camera intended for enabling a target zone to be 
substituted ; 

- Figure 2 shows a common way of mounting a camera head, 
in which tilting movements are added to pan movement (in 
contrast to a so-called "equatorial" mount); 

- Figures 3A and 3B show types of deformation that occur 
when an image is formed; 

Figure 4 shows locations in which the same scene 
reference point is placed for initial intrinsic calibration 
and subsequently for extrinsic calibration; and 

Figure 5 is a simplified summary chart showing one 

possible way in which the mathematical model can be 

implemented . 

DETAILED DESCRIPTION OF THE INVENTION 
The camera shown diagrammatically in Figure 1 comprises a 
stand 10 on which a head support 12 can turn about a pan axis 
14. The head 16 can turn relative to the support 12 about a 
tilt axis 18. On the camera head there is mounted a lens 20 
which generally, carries a focusing ring 22 and a zoom ring 24. ' 
These two rings are provided with respective absolute angle 
encoders of high resolution. The tilt angle (f> relative to the 
support is provided by a sensor 26. The angle 9 about the pan 
axis is supplied by a sensor 30. All of these sensors are 
connected to acquisition electronics 32 which converts the 
data form the sensors into the form of a serial signal (RS232, 
RS422) or into audio modulation. 

In order to ensure that inlaying can be performed 
precisely, it is necessary to use the signals from the sensors 
to determine exactly where the target zone for replacement is 
to be found in the image, which zone remains motionless within 
the scene, and it is necessary to do so regardless of camera 
zoom, focus, or orientation. Unfortunately, image 

transformations when these parameters are combined are far 
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from being simple homothetic transforms, and they are far from 
being the same for all lenses of a same type. 

Calibration is a deterministic process, thus making it 
possible to store the behavior of the lens for. all zoom and 
focus positions. 

The following are described below in succession: 

- intrinsic calibration: this generates a file containing 
the intrinsic geometrical characteristics of the lens; the 
lens is then entered into a database of calibrated lenses ,- 
calibration needs to be done for each lens, even for lenses of 
the same model and from the same manufacturer, but on each 
lens it need be done only once and for always; and 

extrinsic calibration known as " lens camera setting 
(LCS) : it takes account of operating constraints associated 
with the lens being mounted and dismounted on a particular 
camera giving rise to alignment differences which occur with a 
single camera and with different cameras (image format) . LCS 
needs to be performed each time a lens is mounted on a camera, 
even if the same lens is newly mounted on the same camera 
(mechanical tolerances) . 

Once operations have been performed, the behavior of the 
camera used for shooting will have been stored for an entire 
series of values delivered by the zoom sensor Z and the focus- 
sensor F, thus making interpolation possible. 

Intrinsic calibration, transposable from one camera to another 

The lens is modelled by defining a plurality of parameters 
that vary as a function of its focal length f at least: 

- absolute focal length for two known distance values of 
the target points, one being at infinity PI* and the other 
point, P2 , being nearby (in the range 5 m to 15 m, depending 
on the type of lens) ; 

- a coefficient K representing geometrical deformation in 
the radial direction; and 

- a focal length correction associated with the focus 
setting ; 

and global parameters that are independent of zoom: 
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- position of the eye; 

- aspect ratio (pixel height/width ratio in the matrix of 
light sensitive sites); and 

- tilt while taking a horizontal shot because the tilt 
5 coder does not give a zero value under such circumstances. 

The position of the eye is a point on the optical axis 
such that if it is caused to coincide with the vertical or pan 
axis, then no parallax is visible in the image when panning 
between a near point (e.g. as implemented on a pane of glass) 
10 and a point at infinity. 

Initialization, wide angle and narrow angle 

The operation begins by entering the identity of the lens 
into memory . 
15 PI and P2 are selected points; 

Fl and F2 are focus values at full zoom Zn (maximum 
magnification or zoom in) selected for PI and P2 ; 

F3 is a focus value selected arbitrarily between 1.1 F2 
and the value of F at the nearest focus distance, Zl, . Zi, 
20 . . . , Zn being reference zoom values; 

T, B, L, R, C, TL, BR, TR, BL being the nine positions 
that a given point is caused to occupy in the image by 
manipulating the camera (Figure 4)', 

the operator begins by performing six initialization 
25 operations. During each such operation, referred to as a 
"snap", the operator moves the camera so as to bring the point 
under observation into one of the nine above-mentioned 
positions, and clicks on the point. On each occasion, the 
values supplied by the coders and the distance between the 
30 points in the scene and the camera are stored: 



PI, 


Fl, 


Zn, 


C 


PI, 


Fl, 


Zl, 


c 


PI, 


Fl, 


Zl, 


L 


PI, 


Fl, 


Zl, 


R 


P2, 


F2, 


Zn, 


C 


PI, 


Fl, 


Zl, 


B 



No computation is performed at the end of this stage; the 
operator then moves onto measurements at wide angle (zoom out) 
Zl and narrow angle (zoom in) Zn . 

Wide and narrow angles, corresponding to different fo cal 
lengths, with focusing Fl, F2 , and F3 

The operator takes five snaps to define the wide angle 
condition: 





PI, 


Fl, 


Zl, 


T 




10 


PI / 


Fl, 


Zl, 


TL 






P2, 


Fl, 


Zl, 


T 






P2, 


F2, 


Zl, 


T 






P2, 


F3, 


Zl, 


T 






and then 


five snaps in narrow 


angle : 


15 


PI , 


Fl, 


Zn, 


H 






PI, 


Fl, 


Zn, 


TL 






P2, 


Fl, 


Zn, 


T 






P2, 


F2, 


Zn, 


T 






P2, 


F3, 


Zn, 


T 




20 


At t 


he 


end 


of this stage, 


the software runs a first series 



of computations, initially giving default values to global 
parameters : 
DO 

{DETERMINE Zl ZOOM PARAMETERS FROM THE 5 Zl SNAPS 
25 DETERMINE Zn ZOOM PARAMETERS FROM THE 5 Zn SNAPS 

DETERMINE GLOBAL PARAMETERS FROM THE 6 D SNAPS 
INITIALIZATION} 
UNTIL CONVERGENCE 

If there is no convergence, the software is designed to 
30 ask the operator to verify the snaps or to do them again. 

The results of all the above snaps and of the following 
snaps are stored in the computer memory in order to enable the 
operator to return to the position of some particular point, 
if the operator so desires. 
35 By way of example, a "snap" can cause a 200 pixel x 150 

pixel window to be scanned in the full image, corresponding to 
about 6 0 k. 



Intermediate zooms 

Measurements are performed at various intermediate amounts 
of zoom Z2, Z3, Zn-1. For each reference zoom, the 

operator takes five more snaps: 

PI, Fl, Zi, T 

PI, Fl, Zi, TL 

P2, Fl, Zi, T 

P2, F2 , Zi, T 

P2, F3, Zi, T 

At the end of this step, the software computes zoom- 
related parameters for each of the values Zi . The software is 
designed to ask the operator to verify the snaps or to start 
again if convergence does not occur. 

The most important point is focal length which varies as a 
function of zoom, and to a lesser extent focus. Focal length 
makes it possible to conserve a relationship between image 
points and coordinates in three-dimensional (3D) space. 

Each computation step makes it possible to define a 
parameter of the lens by a sequence in which: 

- only previous parameters are necessary; 

- parameters that are still unknown do not vary. 
These steps comprise in succession: 

1) determining the optical center of coordinates (Cx, Cy) ' 
on the matrix due to the offset e; 

2) aspect ratio = Ypixel/Xpixel for the reference camera; 

3) absolute focal length, which is a function of zoom for 
constant focus; 

4) focus correction having the form 1 + 8 (Focus) where 
£( Focus) = AxFocus 2 + BxFocus + C; and 

5) the radial distortion coefficient K that is involved in 
a formula of the form r'= r + Kr 3 ; 

where r is the distance to the optical center and K is a 
function solely of the focal length, and is positive for 
barrel deformation and negative for pin-cushion deformation. 

A distinction is made between: 

- constant parameters 

- position of the optical center 

- aspect ratio or "pixel ratio" 



- absolute focal length scale; and 

- parameters that are a function of zoom 

- focal length 

- K 

5 - focus connection 

- eye position. 

In order, the following are determined: 

- the fixed parameters; 

- curves of : 

10 - focal length (zoom) at constant focus; 

- K (of focal length) at constant focus; 

- eye position (focal length) at constant focus; and 

- focus correction (focal length) . 

The analytical expressions found experimentally are piece- 
15 wise logarithmic for zoom over each range between two values 
Zi and they are polynomials of order 2 for focus correction. 
They are stored in the form of particular values for 
parameters in a mathematical model. 

Once intrinsic calibration has been performed, the results 
20 of the processing performed locally or on a remote server are 
stored in the form of a file in a read-only memory (ROM) or in 
a flash memory, or on a hard disk. 

A file is then available which .contains calibration* 
parameters that are as close as possible to the real 
25 characteristics of the lens. 

Extrinsic calibration 

When the lens is mounted on a camera other than the 
reference camera on which its intrinsic characteristics were 

30 determined, there will be a small change in optical center, 
and this gives rise to errors. Camera balancing moves the 
"eye position" and gives rise to a parallax effect (camera 
rotation gives rise to a small amount of movement in 
translation of unknown sign) which is most troublesome when 

35 the object is close. Finally, the image sensor can be of 
geometry and size that are different depending .on image 
standard . 
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Extrinsic calibration, which can also be referred to as 
lens camera setting or LCS , amounts to adjusting calibration 
data. It does no more than recompute constant parameters 
associated with camera initialization, and including 
5 parameters that come from two sources: 

- lens-camera coupling: 

- optical center (Cx, Cy) ; 

- pixel ratio ( Ypixel/Xpixel = pixel aspect ratio) ; 

focal length scale adjustment (absolute focal 

10 length) ; 

base plate offset adjustment (basis for eye 
position) ; and 

- adjustment of absolute focal length scale; 
possibly also camera-head coupling (with multiple 

15 origins for each parameter) : 

- roll: the horizontal axis of the matrix is at an 
angle with the tilt axis, adjustment of the offset of the 
camera baseplate on the stand; and 

- tilt zero: the value given by the tilt encoder when 
20 the optical axis is exactly orthogonal to the pan axis. 

This list is not limiting, and other parameters could be 
incorporated: 

- pan zero (optical axis not exactly parallel to tilt" 
axis) ; 

25 - focus zero (end stop unreliable on some lenses) ; and 

- zoom zero (end stop unreliable on some lenses) . 

In order to be able to provide the computer associated 
with the camera and a subsequent system for inserting a 
virtual model in the position of a target with elements 

30 enabling them to identify the exact location in the image 
where an insertion is to be performed, the on-site operator 
performs above operations 1 and 2 to identify the optical 
center and the aspect ratio and also to determine the absolute 
focal length. The computer containing the program associated 

35 with the lens resets the constant parameters as supplied by 
the file associated with the lens. A mathematical model is 
thus obtained that is of the kind shown in Figure 5 . In other 
words, the information provided by the sensors is corrected so 
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as to enable the exact position of the target in the image as 
reproduced, thus enabling the target to be replaced by the 
model after the model has been scaled appropriately (and 
possibly also deformed due to perspective or to geometrical 
aberrations of the lens), or the model can be added to the 
target, in semi transparent form, after it has been scaled. 



